Method and apparatus for capturing digital video

ABSTRACT

A method and apparatus for capturing digital video includes displaying a preview of a field of view of the imaging device in a user interface of the imaging device. A sequence of images is captured. A main subject and a background in the sequence of images is determined, wherein the main subject is different than the background. A sequence of modified images for use in a final video is obtained, wherein each modified image is obtained by combining two or more images of the sequence of images such that the main subject in the modified image is blur free and the background is blurred. The sequence of modified images is combined to obtain the final video, which is stored in a memory of the imaging device, and displayed in the user interface.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/977,739 filed Oct. 31, 2022, which is a continuation of U.S. patentapplication Ser. No. 17/693,052 filed Mar. 11, 2022, which issued asU.S. as U.S. Pat. No. 11,490,015 on Nov. 1, 2022, which is acontinuation of U.S. patent application Ser. No. 17/514,486 filed Oct.29, 2021, which is a continuation of U.S. patent application Ser. No.17/066,882, filed Oct. 9, 2020, which issued as U.S. Pat. No. 11,165,961on Nov. 2, 2021, which is a continuation of U.S. patent application Ser.No. 16/544,426, filed Aug. 19, 2019, which issued as U.S. Pat. No.10,880,483 on Dec. 29, 2020, which is a continuation of U.S. patentapplication Ser. No. 15/858,339, filed Dec. 29, 2017, which issued asU.S. Pat. No. 10,389,944 on Aug. 20, 2019, which is a continuation ofU.S. patent application Ser. No. 15/431,332, filed Feb. 13, 2017, whichissued as U.S. Pat. No. 9,860,450 on Jan. 2, 2018, which is acontinuation of U.S. patent application Ser. No. 15/149,481, filed May9, 2016, which issued as U.S. Pat. No. 9,800,787 on Oct. 24, 2017, whichis a continuation of U.S. patent application Ser. No. 14/690,818, filedon Apr. 20, 2015, which issued as U.S. Pat. No. 9,338,356 on May 10,2016, which is a continuation of U.S. patent application Ser. No.14/532,654, filed on Nov. 4, 2014, which issued as U.S. Pat. No.9,013,587 on Apr. 21, 2015, which is a continuation of U.S. patentapplication Ser. No. 13/442,370, filed on Apr. 9, 2012, which issued asU.S. Pat. No. 8,922,663 on Dec. 30, 2014, which is a continuation ofU.S. patent application Ser. No. 12/274,032, filed on Nov. 19, 2008,which issued as U.S. Pat. No. 8,154,607 on Apr. 10, 2012, which is acontinuation of U.S. patent application Ser. No. 11/089,081, filed onMar. 24, 2005, which issued as U.S. Pat. No. 8,331,723 on Dec. 11, 2012,which claims the benefit of U.S. Provisional Application Ser. No.60/556,230, filed on Mar. 25, 2004, the contents of each of which areincorporated by reference herein.

FIELD OF INVENTION

The present invention generally relates to digital image processing.

More specifically, this invention relates to processing of digitizedimage data in order to correct for image distortion caused by relativemotion between the imaging device and the subject at the time of imagecapture, or by optical distortion from other sources.

BACKGROUND

When capturing images, as with a camera, it is desirable to captureimages without unwanted distortion. In general, sources of unwanteddistortion can be characterized as equipment errors and user errors.Examples of common equipment errors include inadequate or flawed opticalequipment, and undesirable characteristics of the film or otherrecording media. Using equipment and media of a quality that is suitablefor a particular photograph can help mitigate the problems associatedwith the equipment and the recording medium, but in spite of this, imagedistortion due to equipment errors can still appear.

Another source of image distortion is user error. Examples of commonuser errors include poor image processing, and relative motion betweenthe imaging device and the subject of the image. For example, one commonproblem that significantly degrades the quality of a photograph is theblur that results from camera movement (i.e. shaking) at the time thephotograph is taken. This can be difficult to avoid, especially when aslow shutter speed is used, such as in low light conditions, or when alarge depth of field is needed and the lens aperture is small.Similarly, if the subject being photographed is moving, use of a slowshutter speed can also result in image blur.

There are currently many image processing techniques that are used toimprove the quality, or “correctness,” of a photograph. These techniquesare applied to the image either at the time it is captured by a camera,or later when it is post-processed. This is true for both traditional“hardcopy” photographs that are chemically recorded on film, and fordigital photographs that are captured as digital data, for example usinga charged couple device (CCD) or a CMOS sensor. Also, hardcopyphotographs can be scanned and converted into digital data, and arethereby able to benefit from the same digital signal processingtechniques as digital photographs.

Commonly used post-processing techniques for digitally correctingblurred images typically involve techniques that seek to increase thesharpness or contrast of the image. This can give the mistakenimpression that the blur is remedied. However, in reality, this processcauses loss of data from the original image, and also alters the natureof the photograph. Thus, current techniques for increasing the sharpnessof an image do not really “correct” the blur that results from relativemotion between a camera and a subject being photographed. In fact, thedata loss from increasing the sharpness can result in a less accurateimage than the original. Therefore, a different method that actuallycorrects the blur is desirable.

In the prior art, electro-mechanical devices for correcting image blurdue to camera motion are built into some high-quality lenses, variouslycalled “image stabilization”, “vibration reduction”, or similar names bycamera/lens manufacturers. These devices seek to compensate for thecamera/lens movement by moving one or more of the lens elements; hencecountering the effect of the motion. Adding such a device to a lenstypically makes the lens much more expensive, heavier and less sturdy,and can also compromise image quality.

Accordingly, it is desirable to have a technique that corrects fordistortion in photographs without adding excessively to the price,robustness or weight of a camera or other imaging device, or adverselyaffecting image quality.

SUMMARY

The present invention processes image data in order to correct an imagefor distortion caused by imager movement or by movement of the subjectbeing imaged. In another embodiment, the present invention can preventimage distortion due to motion of the imaging device or subject atrelatively slow shutter speeds, resulting in a substantially undistortedimage.

In another embodiment, the present invention measures relative motionbetween the imaging device and the subject by using sensors that detectthe motion. When an image is initially captured, the effect of relativemotion between the imaging device and the subject is that it transformsthe “true image” into a blurred image, according to a 2-dimensionaltransfer function defined by the motion. The invention determines atransfer function that represents the motion and corrects the blur.

In yet another embodiment, the transfer function is estimated usingblind detection techniques. The transfer function is then inverted, andthe inverted function is implemented in an image correcting filter thatessentially reverses the blurring effect of the motion on the image. Theimage is processed through the filter, wherein blur due to the motion isreversed, and the true image is recovered.

In yet another embodiment, the invention uses the transfer function tocombine consecutive images taken at a fast shutter speed to avoid blurdue to motion between camera and subject that could result from using aslow shutter speed. In still another embodiment, the image sensor ismoved to counter camera motion while the image is being captured.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a portion of memory having memory locations wherein elementsof a recorded image are stored.

FIG. 2 is a portion of memory having memory locations wherein elementsof a deconvolution filter are stored.

FIG. 3 is a portion of memory having memory locations wherein therecorded image is stored for calculating the next value of a correctedimage.

FIG. 4 is a functional block diagram of a system for correcting an imagefor distortion using a transfer function representing the distortion,wherein the transfer function is derived from measurements of the motionthat caused the distortion.

FIG. 5 is a functional block diagram of a system for correcting an imagefor distortion using a transfer function representing the distortion,wherein the transfer function is derived using blind estimationtechniques.

FIG. 6 shows a unit for iterative calculation of the corrective filtercoefficients and estimation of the correct image data.

FIG. 7 illustrates support regions of an image r(n,m) and of a transferfunction h(n,m), and the transfer function h(n,m) being applied todifferent parts of the image r(n,m).

FIG. 8 shows a unit for blind deconvolution to calculate the correctimage data.

FIG. 9 is an image of an object being captured on an image sensorwherein pixel values represent points of the image.

FIG. 10 illustrates the effect of moving an imager while capturing animage, resulting in multiple copies of the image being recorded overeach other, causing blur.

FIG. 11 illustrates combining images taken at fast shutter speeds toresult in the equivalent of a final image taken at a slower shutterspeed, but with reduced blur.

FIG. 12 illustrates image blur correction where an image sensor is movedto compensate for imager movement.

FIG. 13 is an example of an image distorted by movement of the imagerwhen the image was captured.

FIG. 14 represents the image of FIG. 13 corrected according to thepresent invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention will be described with reference to the figureswherein like numerals represent like elements throughout. Although theinvention is explained hereinafter as a method of correcting for imagedistortion due to the shaking of a camera when a picture is taken,similar distortions can also be caused by other types of imagingequipment and by imperfections in photo processing equipment, movementof the subject being photographed, and other sources. The presentinvention can be applied to correct for these types of distortions aswell. Additionally, although reference is made throughout thespecification to a camera as the exemplary imaging device, the presentinvention is not limited to such a device. As aforementioned, theteachings of the present invention may be applied to any type of imagingdevice, as well as image post-processing techniques.

Capturing and recording a photograph, for example by a camera, involvesgathering the light reflected or emanating from a subject, passing itthrough an optical system, such as a series of lenses, and directing itonto a light sensitive recording medium. A typical recording medium intraditional analog photography is a film that is coated with lightsensitive material. During processing of the exposed film, the image isfixed and recorded. In digital cameras, the recording medium istypically a dense arrangement of light sensors, such as a Charge-CoupledDevice (CCD) or a CMOS sensor.

The recording medium continuously captures the impression of the lightthat falls upon it as long as the camera shutter is open. Therefore, ifthe camera and the subject are moving with respect to each other (suchas in the case when the user is unsteady and is shaking the camera, orwhen the subject is moving), the recorded image becomes blurred. Toreduce this effect, a fast shutter speed can be used, thereby reducingthe amount of motion occurring while the shutter is open. However, thisreduces the amount of light from the subject captured on the recordingmedium, which can adversely affect image quality. In addition,increasing the shutter speed beyond a certain point is not alwayspractical. Therefore, undesired motion blur occurs in many picturestaken by both amateur and professional photographers.

The nature of the blur is that the light reflected from a referencepoint on the subject does not fall on a single point on the recordingmedium, but rather it ‘travels’ across the recording medium. Thus, aspread-out, or smudged, representation of the reference point isrecorded.

Generally, all points of the subject move together, and the optics ofthe camera and the recording medium also move together. For example, inthe case of a photograph of a moving car, wherein an image of the car isblurred due to uniform motion of all parts of the car. In other words,the image falling on the recording medium ‘travels’ uniformly across therecording medium, and all points of the subject blur in the same manner.

The nature of the blur resulting from uniform relative motion can beexpressed mathematically. In a 2-dimensional space with discretecoordinate indices ‘n’ and ‘in’, the undistorted image of the subjectcan be represented by s(n,m), and a transfer function h(n,m) can be usedto represent the blur. Note that h(n,m) describes the way the image‘travels’ on the recording medium while it is captured. The resultingimage that is recorded, r(n,m), is given by:r(n,m)=s(n,m)**h(n,m);  Equation (1)where ** represents 2-dimensional convolution. The mathematicaloperation of convolution is well known to those skilled in the art anddescribes the operation:

$\begin{matrix}{{{r\left( {n,m} \right)} =},{\sum\limits_{i = {- \infty}}^{\infty}{\sum\limits_{j = {- \infty}}^{\infty}{{h\left( {i,j} \right)}{{s\left( {{n - i},{m - j}} \right)}.}}}}} & {{Equation}(2)}\end{matrix}$

In the sum operations in Equation (2), the summation limits areinfinite. In practice, the summations are not infinite, since thesupport region of the transfer function is finite. In other words, theregion where the function is non-zero is limited by the time the camerashutter is open and the amount of motion. Therefore, the summation iscalculated for only the indices of the transfer function where thefunction itself is non-zero, for example, from i=−N . . . N and j=—M . .. M.

If the transfer function h(n,m) is known, or its estimate is available,the blur that it represents can be “undone” or compensated for in aprocessor or in a computer program, and a corrected image can beobtained, as follows. Represent the “reverse” of the transfer functionh(n,m) as h⁻¹(n,m) such that:h(n,m)**h ⁻¹(n,m)=δ(n,m);  Equation (3)where δ(n,m) is the 2-dimensional Dirac delta function, which is:

$\begin{matrix}{{\delta\left( {n,m} \right)} = \left\{ {\begin{matrix}{{1{if}\ n} = {m = 0}} \\{0{otherwise}}\end{matrix}.} \right.} & {{Equation}(4)}\end{matrix}$

The delta function has the property that when convolved with anotherfunction, it does not change the nature of that function. Therefore,once h(n,m) and hence h⁻¹(n,m) are known, an image r(n,m) can be putthrough a correcting filter, called a “deconvolution filter”, whichimplements the inverse transfer function w(n,m)=h⁻¹(n,m) and undoes theeffect of blur. Then:

$\begin{matrix}\begin{matrix}{{{r\left( {n,m} \right)}^{**}{w\left( {n,m} \right)}} = {{r\left( {n,m} \right)}^{**}{h^{- 1}\left( {n,m} \right)}}} \\{= {{s\left( {n,m} \right)}^{**}{h\left( {n,m} \right)}^{**}{h^{- 1}\left( {n,m} \right)}}} \\{= {{s\left( {n,m} \right)}^{**}{\delta\left( {n,m} \right)}}} \\{{= {s\left( {n,m} \right)}};}\end{matrix} & {{Equation}(5)}\end{matrix}$and the correct image data s(n,m) is recovered.

The deconvolution filter in this example is such that:

$\begin{matrix}{{\sum\limits_{i = {- N}}^{N}{\sum\limits_{j = {- M}}^{M}{{w\left( {i,j} \right)}{h\left( {{n - i},{m - j}} \right)}}}} = \left\{ {\begin{matrix}{{1\ {if}\ n} = {m = 0}} \\{0{otherwise}}\end{matrix}.} \right.} & {{Equation}(6)}\end{matrix}$Because of the property that the deconvolution operation forces theoutput of the convolution to be zero for all but one index, this methodis called the “zero-forcing algorithm”. The zero-forcing algorithmitself is but one method that can be used, but there are others possiblealso, such as the least mean-square algorithm described in more detailbelow.

In order to define a deconvolution filter, the transfer function h(n,m)representing the relative motion between the imager and the subject mustbe derived from measuring the motion, or alternatively by using blindestimation techniques. The inverse function h⁻¹(n,m) must then becalculated and incorporated in a filter to recover a corrected images(n,m). It is possible to determine h(n,m) using sensors that detectmotion, and record it at the time the image is captured.

One embodiment of the present invention includes one or more motionsensors, attached to or included within the imager body, the lens, orotherwise configured to sense any motion of the imager while an image isbeing captured, and to record this information. Such sensors arecurrently commercially available which are able to capture movement in asingle dimension, and progress is being made to improve their accuracy,cost, and characteristics. To capture motion in two dimensions, twosensors may be used, each capable of detecting motion in a singledirection. Alternatively, a sensor able to detect motion in more thanone dimension can be used.

The convolution in Equation (5) can be performed using memory elements,by performing an element-by-element multiplication and summation overthe support region of the transfer function. The recorded image isstored, at least temporarily, in memory elements forming a matrix ofvalues such as shown in FIG. 1 . Similarly, the deconvolution filterw(n,m) is stored in another memory location as shown in FIG. 2 . Thedeconvolution operation is then performed by multiplying the values inthe appropriate memory locations on an element-by-element basis, such asmultiplying r(n,m) and w(0,0); r(n−1,m) and w(1,0), and so on, andsumming them all up.

Element-by-element multiplication and summing results in theconvolution:

$\begin{matrix}{{y\left( {n,m} \right)} = {\sum\limits_{i = {- N}}^{N}{\sum\limits_{j = {- M}}^{M}{{w\left( {i,j} \right)}{{r\left( {{n - i},{m - j}} \right)}.}}}}} & {{Equation}(7)}\end{matrix}$To calculate the next element, y(n+1,m) for example, the deconvolutionfilter w(n,m) multiplies the shifted memory locations, such as shown inFIG. 3 , followed by the summation. Note that the memory locations donot need to be shifted in practice; rather, the pointers indicating thememory locations would move. In FIG. 1 and FIG. 3 portions of r(n,m) areshown that would be included in the element-by-element multiplicationand summation, and this portion is the same size as w(n,m). However, itshould be understood that r(n,m), that is the whole image, is typicallymuch larger than the support region of w(n,m). To determine value of theconvolution for different points, an appropriate portion of r(n,m) wouldbe included in the calculations.

The filter defined by Equation (5) is ideal in the sense that itreconstructs the corrected image from the blurred image with no dataloss. A first embodiment calculates the inverse of h(n,m) where h(n,m)is known. As explained above, by making use of motion detecting devices,such as accelerometers, the motion of the imager (such as a cameraand/or the associated lens) can be recorded while the picture is beingcaptured, and the motion defines the transfer function describing thismotion.

A functional block diagram of this embodiment in accordance with thepresent invention is illustrated in FIG. 4 , wherein a method 40 forcorrecting image distortion is shown. An image r(n,m) from camera opticsis captured by an imager (step 41) and recorded in memory (step 42).Simultaneously, motion sensors detect and record camera motion (step 43)that occurs while the shutter of the camera is open. The transferfunction representing the motion h(n,m) is derived (step 44), and theinverse transfer function h⁻¹(n,m) is determined (step 46). The inversetransfer function is applied in a corrective filter (step 48) to theimage, which outputs a corrected image s(n,m) (step 49).

In this and other embodiments that make use of motion sensors torepresent the imager's movement, derivation of the transfer functionfrom motion information (step 44) considers the configuration of theimager and the lens also. For an imager that is a digital camera, forexample, the focal length of the lens factors into the way the motion ofthe imager affects the final image. Therefore, the configuration of theimager is part of the derivation of h(n,m). This is important especiallyfor imagers with varying configurations, such as digital cameras withinterchangeable lenses.

In this first embodiment of the invention, an iterative procedure isused to compute the inverse transfer function from h(n,m). Theapproximate inverse transfer function at iteration k is denoted as ĥ_(k)⁻¹(n,m). At this iteration, output of the deconvolution filter is:

$\begin{matrix}\begin{matrix}{{y_{k}\left( {n,m} \right)} = {{{\overset{\hat{}}{h}}_{k}^{- 1}\left( {n,m} \right)}^{**}{r\left( {n,m} \right)}}} \\{= {\sum\limits_{i}{\sum\limits_{j}{{{\overset{\hat{}}{h}}_{k}^{- 1}\left( {i,j} \right)}{{r\left( {{n - i},{m - j}} \right)}.}}}}}\end{matrix} & {{Equation}(8)}\end{matrix}$

The filter output can be written as the sum of the ideal term and theestimation noise as:

$\begin{matrix}\begin{matrix}{{y_{k}\left( {n,m} \right)} = {{{h^{- 1}\left( {n,m} \right)}^{**}{r\left( {n,m} \right)}} +}} \\{\left( {{{\overset{\hat{}}{h}}_{k}^{- 1}\left( {n,m} \right)} - {h^{- 1}\left( {n,m} \right)}} \right)^{**}{r\left( {n,m} \right)}} \\{{= {{s\left( {n,m} \right)} + {v_{k}\left( {n,m} \right)}}};}\end{matrix} & {{Equation}(9)}\end{matrix}$where v(n,m) is the estimation noise which is desirable to eliminate. Aninitial estimate of the correct image can be written as:ŝ _(k)(n,m)=ĥ _(k) ⁻¹(n,m)**r(n,m).  Equation (10)

However, this estimate can in general be iteratively improved. There area number of currently known techniques described in estimation theory toachieve this. A preferable option is the Least Mean-Square (LMS)algorithm. A block diagram of a calculation unit 60 which implementsthis method is shown in FIG. 6 .

As an initial state, ĥ⁻¹ ₀(n,m) is set to equal μr(n,m). Then, thefollowing steps are iteratively repeated:

Step 1, an estimate of the correct image is calculated in a first2-dimensional finite impulse response (2D FIR) filter 62:ŝ _(k)(n,m)=ĥ _(k) ⁻¹(n,m)**r(n,m).

Step 2, a received signal based on the estimated correct image iscalculated in a second 2D FIR filter 64:{tilde over (r)} _(k)(n,m)=ŝ _(k)(n,m)**h(n,m);and the estimation error is calculated using an adder 66:e _(k)(n,m)=r _(k)(n,m)−{tilde over (r)} _(k)(n,m).

Step 3, the inverse transfer function coefficients are then updated inthe LMS algorithm unit 68:ĥ _(k+1) ⁻¹(n,m)=ĥ _(k) ⁻¹(n,m)+μr(n,m)e _(k)(n,m);where μ is the step-size parameter.

These steps are repeated until the estimation error becomes small enoughto be acceptable; which value can be predetermined or may be set by auser. As the iterative algorithm converges, the estimated inversetransfer function approaches the correct inverse transfer functionh⁻¹(n,m). The inverse transfer function coefficients are thecoefficients of the deconvolution filter, and the estimate ŝ(n,m)converges to s(n,m), the correct image, at the same time.

This process can be repeated for the entire image, but it is lesscomplex, and therefore preferable, to find the inverse filter first overa single transfer function support region, then apply it to the entireimage r(n,m).

While the above Steps 1-3 are being repeated, a different portion of therecorded image r(n,m) can be used in each iteration. As in FIG. 7 , itshould be noted that the recorded image r(n,m) typically has a muchlarger support region than the transfer function h(n,m) that representsthe camera motion. Therefore, the above steps are preferably performedover a support region of h(n,m), and not over the entire image r(n,m),for each iteration.

Although the present invention has been explained with reference to theLMS algorithm, this is by way of example and not by way of limitation.It should be clear to those skilled in the art that there are otheriterative algorithms beside the LMS algorithm that can be used toachieve acceptable results, and also that there are equivalent frequencydomain derivations of these algorithms. For example, it is possible towrite Equation (1) in frequency domain as:R(ω₁,ω₂)=S(ω₁,ω₂)H(ω₁,ω₂);  Equation (11)where R(ω₁,ω₂), S(ω₁,ω₂), and H(ω₁,ω₂) are the frequency domainrepresentations (Fourier Transforms) of the captured image, the correctimage, and the transfer function, respectively, and therefore:

$\begin{matrix}{{S\left( {\omega_{1},\omega_{2}} \right)} = {\frac{R\left( {\omega_{1},\omega_{2}} \right)}{H\left( {\omega_{1},\omega_{2}} \right)}.}} & {{Equation}(12)}\end{matrix}$

To obtain s(n,m) one would calculate S(ω₁,ω₂) as above and take theInverse Fourier Transform, which should be known to those skilled in theart. However, this method does not always lead to well behavedsolutions, especially when numerical precision is limited.

In a second embodiment of the present invention, h(n,m) is not known.This second embodiment uses so-called blind deconvolution, whereby thetransfer function h(n,m) is estimated using signal processingtechniques. A functional block diagram of this embodiment is illustratedin FIG. 5 , wherein a method 50 for correcting image distortionaccording to this embodiment is shown. An image r(n,m) from the opticsfrom a camera is captured (step 51) and recorded in memory (step 52).Unlike the first embodiment, there are no motion sensors to detect andrecord camera motion that occurs while the shutter of the camera isopen. Instead, the transfer function representing the motion h(n,m) isderived using blind estimation techniques (step 54), and the inversetransfer function h⁻¹(n,m) is determined (step 56). The inverse transferfunction is applied in a corrective filter to the image (step 58), whichoutputs a corrected image s(n,m) (step 59).

Blind equalization techniques are used to obtain the deconvolutionfilter coefficients. This is also an iterative LMS algorithm, similar tothat used in the first embodiment. In this second embodiment, aniterative procedure is also used to compute an approximate deconvolutionfilter, and the approximation is improved at each iteration until itsubstantially converges to the ideal solution. As aforementioned withrespect to the first embodiment, the level of convergence may bepredetermined or may be set by a user. The approximate deconvolutionfilter is denoted at iteration k as ŵ_(k) (n,m). At this iteration, theoutput of the deconvolution filter is:

$\begin{matrix}\begin{matrix}{{y_{k}\left( {n,m} \right)} = {{{\overset{\hat{}}{w}}_{k}\left( {n,m} \right)}^{**}{r\left( {n,m} \right)}}} \\{{= {\sum{\sum{{{\overset{\hat{}}{w}}_{k}\left( {i,j} \right)}{r\left( {{n - i},{m - j}} \right)}}}}};}\end{matrix} & {{Equation}(13)}\end{matrix}$

The filter output can be written as the sum of the ideal term and theestimation noise as:

$\begin{matrix}\begin{matrix}{{y_{k}\left( {n,m} \right)} = {{{w\left( {n,m} \right)}^{**}{r\left( {n,m} \right)}} +}} \\{\left\lbrack {{{\overset{\hat{}}{w}}_{k}\left( {n,m} \right)} - {w\left( {n,m} \right)}} \right\rbrack^{**}{r\left( {n,m} \right)}} \\{{= {{s\left( {n,m} \right)} + {v_{k}\left( {n,m} \right)}}};}\end{matrix} & {{Equation}(14)}\end{matrix}$where v(n,m) is the estimation noise, which is desirable to eliminate.An initial estimate of the correct image can be written as:ŝ _(k)(n,m)=ŵ _(k)(n,m)**r(n,m).  Equation (15)

However, this estimate can be iteratively improved. There are a numberof currently known techniques described in estimation theory to achievethis. A preferable option is the LMS algorithm. A block diagram of acalculation unit 80 which implements this method is shown in FIG. 8 .

As an initial state, ĥ⁻¹ ₀(n,m) is set equal to μr(n,m). Then, thefollowing steps are iteratively repeated:

Step 1, an estimate of the correct image is calculated in a first 2D FIRfilter 82:ŝ _(k)(n,m)=ĥ _(k) ⁻¹(n,m)**r(n,m).

Step 2, a received signal based on the estimated correct image iscalculated in a non-linear estimator 84:{tilde over (r)} _(k)(n,m)=g({circumflex over (s)}(n,m));and the estimation error is calculated using an adder 86:e _(k)(n,m)=r _(k)(n,m)−{tilde over (r)} _(k)(n,m).

Step 3, the inverse transfer function coefficients are then updated inthe LMS algorithm unit 88:ĥ _(k+1) ⁻¹(n,m)=ĥ _(k) ⁻¹(n,m)+μr(n,m)e _(k)(n,m),where μ is the step-size parameter.

The function g(.) calculated in step 2 is a non-linear function chosento yield a Bayes estimate of the image data. Since this function is notcentral to the present invention and is well known to those of skill inthe art, it will not be described in detail hereinafter.

There are known blind detection algorithms for calculating s(n,m) bylooking at higher order statistics of the image data r(n,m). A group ofalgorithms under this category are called Bussgang algorithms. There arealso variations called Sato algorithms, and Godard algorithms. Anotherclass of blind estimation algorithms use spectral properties(polyspectra) of the image data to deduce information about h(n,m). Anyappropriate blind estimation algorithm can be used to determine h(n,m),and to construct a correcting filter.

The first two embodiments of the present invention describedhereinbefore correct blur in an image based on determining a transferfunction that represents the motion of an imager while an image is beingcaptured, and then correcting for the blur by making use of the“inverse” transfer function. One method determines the transfer functionat the time the photograph is being captured by using devices that candetect camera motion directly. The other method generates a transferfunction after the image is captured by using blind estimationtechniques. Both methods then post-process the digital image to correctfor blur. In both cases, the captured image is originally blurred bymotion, and the blur is then removed.

In accordance with a third embodiment of the present invention theblurring of an image is prevented as it's being captured, as describedbelow. When an imager is moved while an image is being captured,multiple copies of the same image are, in effect, recorded over eachother. For example, when an image is captured digitally it isrepresented as pixel values in the sensor points of the image sensor.This is pictorially represented in FIG. 9 , in which the imager (forexample, a camera and its associated lens) are not shown in order tosimplify the depiction.

If the imager is shaken or moved while the image is being captured, thesituation is equivalent to copies of the same image being capturedmultiple times in an overlapping fashion with an offset. The result is ablurred image. This is particularly true if the shutter speed isrelatively slow compared to the motion of the camera. This isgraphically illustrated in FIG. 10 .

When the shutter speed is sufficiently fast compared to the motion ofthe imager, blur does not occur or is very limited because thedisplacement of the imager is not large enough to cause the lightreflected from a point on the image to fall onto more than one point onthe image sensor. This third embodiment of the invention takes advantageof the ability of an imager to record multiple images using fast shutterspeeds. When an image is being captured using a setting of a relativelyslow shutter speed, the imager actually operates at a higher shutterspeed (for instance at the fastest shutter speed at which the imager isdesigned to operate), and captures multiple images “back to back.” Forexample, if the photograph is being taken with a shutter speed settingof 1/125 sec and the fastest shutter speed of the camera is 1/1000 sec,the camera actually captures 8 consecutive images, each taken with ashutter speed setting of 1/1000 sec. Then, the camera combines theimages into a single image by aligning them such that each pixelcorresponding to the same image point in each image is combinedpixel-by-pixel into one-pixel value by adding pixel values, averagingthem, or using any other appropriate operation to combine them. Themultiple images can all be stored and aligned once all of them arecaptured, or alternatively, each image can be aligned and combined withthe first image in “real time” without the need to store all imagesindividually. The blur of the resulting image is substantially reduced,as depicted in FIG. 11 .

The quality of an image can be measured in terms of signal-to-noisepower ratio (SNR). When a fast shutter speed is used, the SNR of theimage is degraded because the image sensor operates less effectivelywhen the amount of light falling on it is reduced. However, sincemultiple images are being added, this degradation is overcome. Indeed,an SNR improvement can be expected using this embodiment, because theimage data is being added coherently while the noise is being addednon-coherently. This phenomenon is the basis for such concepts asmaximal ratio combining (MRC).

To determine how to align the pixel values, a device that can detectmotion, such as an accelerometer or other motion sensor, is attached toor incorporated within the imager, and it records the motion of theimager while the photograph is being taken. The detected motionindicates how much the imager moved while each of the series of imageswas captured, each image having been captured back-to-back with a highshutter speed as explained in the example above. The imager moves eachof the images in the series by an amount which is preferably measured inpixels, in the direction opposite the motion of the imager that occurredduring the interval between the capture of the first image and eachrespective image in the series. Thus, the shift of each image iscompensated for, and the correct pixels are aligned in each of theimages. This is illustrated in FIG. 11 . The combined image will not beblurred since there is no spilling of image points into more than onepixel in the combined final image.

As an alternative to the third embodiment, the reference point foraligning the higher speed images is not the imager location, but thesubject itself. In other words, higher shutter speed images can bealigned and combined such that a designated subject in a field of viewis clear and sharp whereas other parts of the image may be blurred. Forexample, a moving subject such as a car in motion can be the designatedsubject. If high shutter speed images are combined such that the pointsof the image of the moving car are aligned, the image of the car will beclear and sharp, while the background is blurred. As a way to align adesignated subject, such as the car in this example, pattern recognitionand segmentation algorithms may be used that are well known to thoseskilled in the art, and defined in current literature. Alternatively, atracking signal that is transmitted from the subject can be used toconvey its position. Alternatively, the user can indicate, such as by anindicator in a viewfinder, which object in the field of view is thedesignated subject to be kept blur-free.

A fourth embodiment of the invention compensates for movement of theimager or the subject by adjusting the position of the image sensorduring image capture, according to the inverse of the transfer functiondescribing the imager or subject motion, or both. This embodiment isillustrated in FIG. 12 . This embodiment is preferably used in digitalcameras wherein the image sensor 108 is a relatively small component andcan be moved independently of the camera, but can also be used withfilm. Accordingly, this embodiment makes use of motion sensors, anddetects the movement of the camera and/or the subject while the image isbeing captured. The signals from the motion sensors are used to controldevices that adjust the position of the image sensor. In FIG. 12 ,horizontal motion sensor 102 and vertical motion sensor 104 measuremovement of the camera while its shutter (not shown) is open and animage is being captured. The motion information is conveyed to acontroller 106, which determines and sends signals to devices 110 a, 110b, 110 c, and 110 d, which adjust the position of the image sensor 108.The control mechanism is such that the devices 110 a-d, for exampleelectromagnets or servos, move the image sensor 108 in the oppositedirection of the camera motion to prevent motion blur. Additionalsensors (not shown) can be used to detect motion of the subject, and thecontrol mechanism configured to correct for that motion as well.

FIG. 13 shows an example of a photographic image that is blurred due touser movement of the imager while taking the picture. FIG. 14 shows thesame image, corrected according to the present invention. The inventionsubstantially recovers the correct image from the blurred image.

Those skilled in the art will recognize that all embodiments of theinvention are applicable to digitized images which are blurred byuniform motion, regardless of the source of the image or the source ofthe motion blur. It is applicable to digital images blurred due tomotion of the imager, of the subject, or both. In some cases, it is alsoapplicable to images captured on film and then scanned into digitalfiles. In the latter case, however, motion sensor information typicallymay not be available, and therefore only the blind estimation embodimentcan be used. Also, where appropriate, the different embodiments of theinvention can be combined. For example, the superposition embodiment canbe used to avoid most blur, and the correcting filter using blindestimation embodiment can then be applied to correct the combined imagefor any remaining blur.

In describing the invention, no distinction has been made between animager that captures images one at a time, such as a digital camera, andone that captures sequence of images, such as digital or analog videorecorders. A digital video recorder or similar device operatessubstantially the same way as a digital camera, with the addition ofvideo compression techniques to reduce the amount of image data beingstored, and various filtering operations used to improve image quality.The invention is also applicable to digital and analog video capture andprocessing, being applied to each image in the sequence of images, andcan be used in conjunction with compression and other filtering.

The implementation of the apparatus that performs the restoration of theimages to their correct form can be done as part of the imager capturingthe image, or it can be done as a post-process. When done as part of theimager, the image correcting apparatus can be implemented in anintegrated circuit, or in software to run on a processor, or acombination of the two. When done as a post process, a preferredembodiment is that the image data is input into a post processing devicesuch as a computer, and the blind estimation algorithm is performed by acomputer program. In this embodiment, the implementation could be adedicated computer program, or an add-on function to an existingcomputer program.

Where a computer program performs the image restoration, a blindestimation algorithm can be executed by the program to calculate theestimated transfer function h(n,m). Alternatively, motion informationcan be recorded by the camera at the time the image is captured, and canbe downloaded into the program to be used as an input to calculateh(n,m). In either case, the program then derives the correcting filterand applies the filter to correct the image.

It should also be noted that if there are multiple blurred objects in animage, and the blur is caused by the objects moving in differentdirections, the image of each object will be blurred differently, eachblurred object having a different transfer function describing itsmotion. The present invention can allow the user to individually selectindependently blurred parts of the image and individually correct onlythe selected parts, or alternatively, to correct a selected part of theimage at the expense of the rest of the image, resulting in ablur-corrected subject and a blurred background.

When increased accuracy is needed in obtaining h(n,m), those skilled inthe art will recognize that, in some cases, the motion information fromsensors can be used to calculate h(n,m), and an estimate of h(n,m) canalso be calculated by blind estimation and the two transfer functionscan be advantageously combined for more accurate results.

There are other signal processing algorithms and digital filters whichcan be applied to digital images in order to improve their colorsaturation, reduce noise, adjust contrast and sharpness, etc. These canbe incorporated as part of an imager, such as a digital camera, or aspart of a post-processing application, such as a photo editing softwarerunning on a computer. It should be clear to those skilled in the artthat those techniques can be applied in addition to the distortioncorrection of this invention.

What is claimed is:
 1. An imaging apparatus for capturing improveddigital images, comprising: a user interface for displaying a preview ofa field of view of the imaging apparatus, for receiving user input, andfor displaying digital images; a lens comprising plurality of lenselements; an image sensor optically coupled to the lens for capturingdigital images; a memory, and a set of instructions for executing by aprocessor stored in the memory; and a processor communicativelyconnected to the image sensor and to the memory for receiving theinstructions stored in the memory, wherein executing the instructions bythe processor causes the processor to: determine a desired number ofimages; cause the image sensor to capture a desired number of imagesthrough the lens and store the captured images in the memory; receivethe desired number of images captured by the image sensor from thememory; designate one of the images of the desired number of images as areference image; designate an image point in the reference image to beused in aligning the desired number of images; determine an offset foreach of the desired number of images other than the reference image bycomparing a location of the image point in each image relative to alocation of the image point in the reference image; combine the desirednumber of images to calculate pixel values for an improved image bytaking into account the offsets calculated for the images and the pixelvalues in the images being combined; store the improved image in thememory; and display the improved image in the user interface.
 2. Theimaging apparatus of claim 1, wherein the processor causes the imagesensor to capture a desired number of images upon receiving a user inputin the user interface.
 3. The imaging apparatus of claim 1, wherein theprocessor shifts the desired number of images by the offset determinedfor each image in a vertical direction, a horizontal direction, or bothdirections when calculating pixel values for an improved image.
 4. Theimaging apparatus of claim 1, wherein the processor calculates pixelvalues for different parts of the improved image by combining differentimages from among the desired number of images.
 5. The imaging apparatusof claim 1, wherein executing the instructions causes the processor todetermine a shutter speed to be used when capturing the desired numberof images by the image sensor.
 6. The imaging apparatus of claim 1,wherein the processor determines the offsets for the desired number ofimages other than the reference image in number of pixels.
 7. Theimaging apparatus of claim 1, wherein the processor calculates the pixelvalues for the corrected image based on the pixel values in all of thedesired number of -images.
 8. The imaging apparatus of claim 1, whereinthe processor combines the desired number of images selectively bytaking into account only some of the images when calculating pixelvalues for the improved image.
 9. The imaging apparatus of claim 1,wherein the processor further applies an image filter to the improvedimage.
 10. The imaging apparatus of claim 1, wherein the processorcalculates pixel values for the improved image by taking into accountthe pixel values in the combined images equally.
 11. A method for use inan imaging apparatus for capturing improved digital images, comprising:displaying a preview of a field of view of the imaging apparatus in auser interface; determining a desired number of images; capturing adesired number of images using an image sensor of the imaging apparatusand a lens of the imaging apparatus, and store the captured images in amemory of the imaging apparatus; designating one of the images of thedesired number of images as a reference image; designating an imagepoint in the reference image to be used in aligning the desired numberof images; determining an offset for each of the desired number ofimages other than the reference image by comparing a location of theimage point in each image relative to a location of the image point inthe reference image; combining the desired number of images to calculatepixel values for an improved image by taking into account the offsetscalculated for the images and the pixel values in the images beingcombined; storing the improved image in the memory; and displaying theimproved image in the user interface.
 12. The method of claim 11,wherein a desired number of images are captured by the image sensor inresponse to the imaging apparatus receiving a user input in the userinterface.
 13. The method of claim 11, wherein calculating pixel valuesfor an improved image includes shifting the desired number of images bythe offset determined for each image in a vertical or a horizontaldirection.
 14. The method of claim 11, wherein pixel values fordifferent parts of the improved image are calculated by combiningdifferent images from among the desired number of images.
 15. The methodof claim 11, wherein the method further includes determining a shutterspeed to be used when capturing the desired number of images by theimage sensor.
 16. The method of claim 11, wherein the offsets for thedesired number of images other than the reference image are determinedin number of pixels.
 17. The method of claim 11, wherein pixel valuesfor the corrected image are calculated based on the pixel values in allof the desired number of images.
 18. The method of claim 11, wherein thedesired number of images are combined selectively by taking into accountonly some of the images when calculating pixel values for the improvedimage.
 19. The method of claim 11, wherein the method includes applyingfurther filtering to the improved image.
 20. The method of claim 11,wherein pixel values for the improved image are calculated by takinginto account the pixel values in the combined images equally.